Information display system, information display apparatus and non-transitory storage medium

ABSTRACT

The matching processor acquires the key information such as position information and/or the like by the key information acquirer, and notifies to an external apparatus via the communicator. The matching processor stores the acquired key information in the storer corresponding to the identification information which is acquired from the external apparatus. When the acquired identification information does not exist in a AR data management table, the matching processor acquires the AR data by notifying the identification information to the external apparatus. When there is no empty record in a key information management table, key information of which usage date and time is old is considered as a deletion target. Moreover, when there is no empty record in the AR data management table, the AR data of which usage date and time is old is considered as the deletion target.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of Japanese Patent Application No.2010-272834, filed on Dec. 7, 2010, the entire disclosure of which isincorporated by reference herein.

FIELD

This application relates generally to an information display system, aninformation display apparatus and a non-transitory storage medium, andmore particularly, to the information display system, the informationdisplay apparatus and the non-transitory storage medium suitable forinformation display using AR technology.

BACKGROUND

A high-performance portable communication terminal provided with animaging function, such as so-called smartphone, is becoming popular and,a service (hereinafter referred to as “AR service”) which is applied theAR (Augmented Reality) technology to such terminal device is realized.In the AR service, since additional information is superimposed on thephotographed image captured by the terminal in real time, variousinformation may be easily obtained in a place of picking up, as NationalPatent Publication No. 2008-510254, for example.

In a case of superimposing information on the captured image and in acase where there are a plurality of subjects in the image, it isrequired a position alignment for correctly associating the informationto be superimposed and each subject. As one method of the positionalignment, for example, Unexamined Japanese Patent Application KOKAIPublication No. 2009-230255 discloses that the subject is specified byperforming image recognition on the captured image and applying matchingwith various captured images accumulated in a server or a database on anetwork.

According to this method, various things become the targets of the ARservice and, the number of times of inquiry to a server increases as theAR service is better. Moreover, a huge amount of reference images willbe accumulated in a database and, the quantity of the informationprovided in response to the inquiry also increases, thus, networktraffic load will be increased.

Moreover, in a terminal which uses the provided information, since thequantity of acquired information also increases memory usage increasesand, it takes time for the matching due to increasing the number ofmatching objects. As a result, the performance at the time of using theAR service would be declined, and convenience would be decreased.

The present disclosure is made in view of the above mentioned actualcondition, and an object of the present disclosure is to provide aninformation display system, an information display apparatus and anon-transitory storage medium which can improve the convenience in theAR service.

SUMMARY

According to a first aspect of the present disclosure, an informationdisplay system for displaying information comprising:

an information display apparatus which includes an imaging unit; and

an information provision apparatus;

wherein the information display apparatus comprises:

-   -   a key information acquiring unit which acquires key information        corresponding to a subject in a captured image captured by the        imaging unit; and    -   a key information notifying unit which notifies the key        information to the information provision apparatus,

wherein the information provision apparatus comprises:

-   -   an AR data storage unit which stores augmented reality (AR) data        corresponding to identification information for identifying the        AR data and key information notified by the key information        notifying unit, the AR data including (i) a reference image        including a reference subject, and (ii) superimposing        information corresponding to the reference image; and    -   an information providing unit (i) which transmits the        identification information corresponding to the key information        to the information display apparatus when the key information is        provided from the information display apparatus, and (ii) which        transmits the AR data corresponding to the identification        information to the information display apparatus when the        identification information is provided from the information        display apparatus,

wherein the information display apparatus further comprises:

-   -   an identification information acquiring unit which acquires the        identification information provided from the information        provision apparatus in accordance with a notification of the key        information;    -   a key information recording unit which records the corresponding        the key information and the identification information in a        first table defined by the number of records;    -   an identification information notifying unit which notifies the        identification information recorded in the first table to the        information provision apparatus;    -   an AR data acquiring unit which acquires the AR data provided        from the information provision apparatus in accordance with a        notification of the identification information;    -   an AR data recording unit which records acquired AR data        corresponding to the first table in a second table defined by        the number of records; and    -   an AR data displaying unit which superimposes and displays the        superimposing information corresponding to the reference image        on the captured image based on a result from matching the        subject in the captured image with the reference image included        in the AR data which is recorded by the AR data recording unit.

According to a second aspect of the present disclosure, an informationdisplay apparatus comprising:

an imaging unit;

a communication unit which communicates with an external apparatus;

a displaying unit;

a key information acquiring unit which acquires key informationcorresponding to a subject in a captured image captured by the imagingunit;

a key information notifying unit which notifies the key informationacquired by the key information acquiring unit to the external apparatusthrough a communication network;

an identification information acquiring unit which acquires theidentification information indicating augmented reality (AR) datatransmitted by the external apparatus in accordance with a notificationof the key information by the key information notifying unit through thecommunication network, the AR data including (i) a reference imageincluding a reference subject, and (ii) superimposing informationcorresponding to the reference image;

a key information recording unit which records the key informationacquired by the key information acquiring unit corresponding to a usagedate and time of the key information and identification informationacquired by the identification information acquiring unit in a firsttable defined the number of records;

an identification information notifying unit which notifies theidentification information recorded in the first table to the externalapparatus through the communication network;

an AR data acquiring unit which acquires the AR data transmitted by theexternal apparatus in accordance with a notification of theidentification information by the identification information notifyingunit; and

an AR data recording unit which records the AR data acquired by the ARdata acquiring unit, which corresponds to notified identificationinformation and the usage date and time of the AR data, in a secondtable defined by the number of records;

wherein the displaying unit has

-   -   an AR data displaying unit which superimposes and displays the        superimposing information corresponding to the reference image        on the captured image based on a result from matching the        subject in the captured image with the reference image included        in the AR data which is recorded in the second table,

wherein

the identification information notifying unit notifies theidentification information to the external apparatus when theidentification information recorded in the first table is not recordedin the second table;

the key information recording unit executes a deletion based on theusage date and time of the key information already recorded, when noempty record is present in the first table at the time the keyinformation acquiring unit acquires the key information; and

the AR data recording unit executes a deletion based on the usage dateand time of the AR data already recorded, when no empty record ispresent in the second table at the time the identification informationnotifying unit notifies the identification information.

According to a third aspect of the present disclosure, a non-transitorycomputer-readable storage medium which stores a program for causing acomputer of an information display apparatus including an imaging unit,a communicating unit which communicates with an external apparatus and adisplaying unit to perform functions comprising:

acquiring key information corresponding to a subject in a captured imagecaptured by the imaging unit;

notifying the key information to the external apparatus through acommunication network;

acquiring the identification information indicating augmented reality(AR) data transmitted by the external apparatus in accordance with anotification of the key information through the communication network,the AR data including (i) a reference image including a referencesubject and (ii) superimposing information corresponding to thereference image;

recording acquired key information corresponding to a usage date andtime of the key information and acquired identification information in afirst table defined by the number of records;

notifying the identification information to the external apparatusthrough the communication network when the identification informationrecorded in the first table is not recorded in a second table recordingthe corresponding the identification information and the AR dataacquired from the external apparatus;

acquiring the AR data transmitted by the external apparatus inaccordance with a notification of the identification information;

recording acquired AR data corresponding to notified identificationinformation and the usage date and time of the AR data in the secondtable defined by the number of records;

superimposing and displaying the superimposing information on thecaptured image on the displaying unit using the AR data recorded in thesecond table;

executing a deletion based on the usage date and time of the keyinformation having already recorded, when no empty record is present inthe first table at the time the key information is acquired; and

executing a deletion based on the usage date and time of the AR datahaving already recorded, when no empty record is present in the secondtable at the time notifying the identification information to theexternal apparatus.

According to the present disclosure, it is possible to improve theconvenience at the time of utilizing a service using the AR technology.

The above and further objects and novel features of the presentdisclosure will more fully appear from the following detaileddescription when the same is read in conjunction with the accompanyingdrawings. It is to be expressly understood, however, that the drawingsare for the purpose of illustration only and are not intended as adefinition of the limits of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of this application can be obtained whenthe following detailed description is considered with reference to thefollowing drawings, in which:

FIG. 1 is a schematic diagram showing an information display systemconcerning embodiments of the present disclosure;

FIGS. 2A and 2B are schematic diagrams showing configuration of externalappearance of the information display apparatus shown in FIG. 1, FIG. 2Ashows the configuration of external appearance on a main surface sideand, FIG. 2B shows the configuration of external appearance on a backsurface side;

FIG. 3 is a block diagram showing a configuration of the informationdisplay apparatus shown in FIG. 1;

FIG. 4 is a functional block diagram showing functions which arerealized by the controller shown in FIG. 3;

FIG. 5 is a block diagram showing a configuration of the informationprovision apparatus shown in FIG. 1;

FIG. 6 is a functional block diagram showing functions which arerealized by the controller shown in FIG. 5;

FIG. 7 is a diagram showing an example of a “reference image storingtable” which is registered in the storer shown in FIG. 5;

FIG. 8 is a flowchart for explaining “AR display process (1)” of a firstembodiment of the present disclosure;

FIG. 9 is a flowchart for explaining “AR information provision process(1)” of the first embodiment of the present disclosure;

FIGS. 10A and 10B are diagrams for explaining an operation concerningthe first embodiment of the present disclosure, FIG. 10A shows anexample of a scene assumed, and FIG. 10B shows examples of referenceimages to be used;

FIGS. 11A and 11B are diagrams showing examples of screen displaysconcerning the “AR display process (1)” shown in FIG. 8, FIG. 11A showsthe example of a distribution display, and FIG. 11B shows the examplewhich displays a guide on the distribution display;

FIGS. 12A and 12B are diagrams showing examples of screen displaysconcerning the “AR display process (1)” shown in FIG. 8, FIG. 12A showsa display example in a case where the guide moves, and FIG. 12B showsthe example of displaying superimposing information;

FIGS. 13A to 13C are diagrams showing other examples of the guidedisplay, FIG. 13A shows the example of the guide display which inducesan approach to a subject, FIG. 13B shows the example of the guidedisplay which induces a separation from the subject, and FIG. 13C showsthe example of the guide display which shows a required angle of view;

FIG. 14 is a flowchart for explaining “AR display process (2)”concerning a second embodiment of the present disclosure;

FIG. 15 is a diagram showing an example of a “reference image storingtable” of the second embodiment in the present disclosure;

FIG. 16 shows a flowchart for explaining “key information managementprocess” performed in the “AR display process (2)” shown in FIG. 14;

FIG. 17 shows a flowchart for explaining “AR data management process”performed in the “AR display process (2)” shown in FIG. 14;

FIGS. 18A and 18B are diagrams for explaining a table used in the “ARdisplay process (2)” shown in FIG. 14, FIG. 18A shows an example of a“key information management table”, and FIG. 18B shows an example of the“AR data management table”;

FIG. 19 is a flowchart for explaining a “AR display process (3)”concerning a third embodiment of the present disclosure; and

FIGS. 20A to 20D are diagrams for explaining the reference image and 3Dmodeling data in the “AR display process (3)” shown in FIG. 19, FIG. 20Aexemplarily shows a direction of line of sight to a subject, FIG. 20Bshows an example of the reference image, FIG. 20C shows an example ofgenerated 3D modeling data, and FIG. 20D shows an example of 3D modelingdata after changing the line of sight.

DETAILED DESCRIPTION

The embodiments of the present disclosure will be described below withreference to drawings.

First Embodiment

An information display system 1 of the embodiment of the presentdisclosure will be described with reference to FIG. 1. FIG. 1 is aschematic diagram showing a configuration of the information displaysystem 1 of the embodiment.

As shown in the Figure, the information display system 1 of thisembodiment has an information display apparatus 100 and an informationprovision apparatus 200, and/or the like, which are connected by acommunication network NW.

The communication network NW in this embodiment is WAN (Wide AreaNetwork) including IP network such as Internet and a cellular networkfor mobile communications, for example.

The information display apparatus 100 in this embodiment is a portableterminal device having at least a communication function, an imagingfunction and a displaying function, and so-called smartphone ispreferred for the information display apparatus 100. In this embodiment,the information display apparatus 100 is the smartphone as shown inFIG. 1. In this embodiment, a service which is applied an AR (AugmentedReality) technology (hereinafter, referred to as “AR service”) isrealized by the information display system 1 including the informationdisplay apparatus 100. Specifically, on a captured image which is shotby the imaging function of the information display apparatus 100,information relating to a subject in the image is displayedsuperimposing in real time.

A configuration of the information display apparatus 100 will bedescribed with reference to FIG. 2 and FIG. 3. FIG. 2 is a schematicdiagram showing a configuration of external appearance of theinformation display apparatus 100. FIG. 2A shows a main surface side ofthe information display apparatus 100 and, FIG. 2B shows a back surfaceside of the information display apparatus 100. Moreover, FIG. 3 is ablock diagram showing a configuration of the information displayapparatus 100.

As shown in each of above-mentioned Figures, the information displayapparatus 100 in this embodiment has a controller 110, a communicator120, a key information acquirer 130, an imager 140, aninputter-outputter 150, and a storer 160, and/or the like.

The controller 110 has a CPU (Central Processing Unit), a RAM (RandomAccess Memory), and/or the like. The controller 110 controls eachsection of the information display apparatus 100 and performs eachprocess described later by executing a program.

The communicator 120 is a component for realizing a communicationthrough the communication network NW. For example, The communicator 120is configured by a communication module which can directly access theWAN such as a cellular communication module for CDMA 2000 scheme and acommunication module for IEEE 802.16e (namely, mobile WiMAX) and/or, acommunication module for IEEE 802.11 series (wireless LAN) tocommunicate with an access point of the public wireless LAN and anexternal communication device (a mobile router, and/or the like) whichcan access the WAN.

The key information acquirer 130 is a component for acquiring keyinformation which is required when using the AR service. For example,the key information acquirer 130 is configured by a GPS (GlobalPositioning System) module, a short range wireless communication modulesuch as RFID (Radio Frequency Identification), IrDA (Infrared DataAssociation) and Bluetooth (trademark) and, a visible lightcommunication module. The key information acquirer 130 acquires the keyinformation provided from outside.

The “key information” here includes, for example, position information(latitude and longitude information) which shows a current position ofthe information display apparatus 100, in addition, ID informationprovided to various facilities such as stores which register informationto be superimposed by the AR service in the information provisionapparatus 200, and/or the like. The information display apparatus 100transmits the key information acquired to the information provisionapparatus 200, thereby selecting a reference image which is needed topositioning and information to be provided in the AR service.

Therefore, the GPS module, and/or the like, is used as the component foracquiring the current position information of the information displayapparatus 100. In addition, the short range wireless communicationmodule, the visible light communication module, and/or the like, areused as the component for acquiring the ID information acquirable at thevarious facilities, and/or the like. These components are suitablyconfigured on the external surface of the information display apparatus100 and the inside thereof, for example, as shown in FIG. 2B. However,the information diplay apprataus 100 need not have all these components.The information display apparatus 100 has at least the component foracquiring position information (a GPS module, and/or the like) in thisembodiment.

In this embodiment, the key information acquirer 130 includes at leastthe component for acquiring the current position (a GPS module, and/orthe like) and, the component for acquiring the ID information isarbitrarily included. Moreover, the key information acquirer 130 mayacquire the position information by a method other than the GPS, forexample, the key information acquirer 130 may acquire the positioninformation from a base station accessed in a cellular communication oran access point accessed in wireless LAN. In this case, the keyinformation acquirer 130 includes the communicator 120 to perform thesecommunications.

The imager 140 is a component for realizing the imaging function of theinformation display apparatus 100 and, is configured by a digital cameramodule using a solid-state image sensing device, for example. In thisembodiment, the information provided by the AR service is superimposedon the moving image (photographed image) as live view image obtained bythe imager 140.

In addition, in a case where the above mentioned key information isprovided as visual objects such as a two dimensional code like QR Code(trademark) for example, the key information is acquired by an imagingoperation of the imager 140. In this case, the key information acquirer130 includes the imager 140.

Moreover, as the imager 140, a two lenses type stereo camera as shown inFIG. 2B is preferred. Specifically, the imager 140 is an imaging devicewhich has two lenses arranged in horizontal direction in parallel at aninterval corresponding to parallax. Adapting such stereo camera enablesto perform a ranging to the subject according to the principle oftriangulation and to generate 3D modeling data.

The inputter-outputter 150 includes, at least a display device whichdisplays and output captured image by the imager 140 and informationprovided by the AR service and, an input device which receives inputoperations by a user of the information display apparatus 100. In thisembodiment, a touch panel (touch screen) which integrally realizes thedisplay function and the input function is configured as theinputter-outputter 150. Moreover, the inputter-outputter 150 alsoincludes a speaker and a microphone which are used for voice input andoutput in a telephone call function in the smartphone.

The storer 160 is configured, for example, by a semiconductor memorydevice such as a flash memory and, the storer 160 stores programs to beexecuted by the controller 110, data to be used in various process, datagenerated in the various process and data acquired by operation of thecommunicator 120 and the key information acquirer 130, and/or the like.

Functions which are realized by executing programs stored in the storer160 by the controller 110 will be described with reference to FIG. 4.FIG. 4 is a functional block diagram showing functions which arerealized by the controller 110.

In this embodiment, as shown in the Figure, functions of the ARprocessor 111, the imaging processor 112, and the matching processor113, and/or the like, are realized by the controller 110.

The AR processor 111 performs an application (hereinafter, referred toas “AR application”) which relates to the AR service and is stored inthe storer 160, based on an input signal from the inputter-outputter 150according to user's operation, and/or the like, and controls theoperation thereof. The AR processor 111 performs image processing whichis required for a display operation (hereinafter, referred to as “ARdisplay”) concerning the AR service, and displays on or output to theinputter-outputter 150. In this case, the AR display is performed byperforming the image processing using the capture data, and/or the like,which is stored in the storer 160.

The imaging processor 112 controls the imager 140 to control anoperation relating to the imaging function of the information displayapparatus 100. In this embodiment, the imaging processor 112 controlsthe imaging operation which is required for the AR application performedby the AR processor 111.

The matching processor 113 controls the communicator 120 and the keyinformation acquirer 130 and acquires information from the informationprovision apparatus 200 or various external devices, thereby performinga matching operation which is required for the alignment in the ARdisplay. The key information acquirer 130 is controlled so as to acquirethe key information in accordance with an execution of the ARapplication and, to store the acquired key information in the storer160. Moreover, the communicator 120 is controlled so as to transmit theacquired key information to the information provision apparatus 200,and/or the like, and to acquire the reference image and superimposinginformation which are required for execution of the AR application fromthe information provision apparatus 200. Then, the matching processor113 stores the reference image and superimposing information which areacquired in the storer 160, and compares the captured image acquired bythe imaging processor 112 and the reference image.

Although these functions are logically realized by the controller 110 inthis embodiment, a part of or all of these functions may be realized byhardware, such as ASIC (Application Specific Integrated Circuit), forexample.

Each component of the information display apparatus 100 described aboveis one required to realize the present disclosure and, the othercomponents required to function as the smartphone are arbitrarilyprovided.

Next, the configuration of the information provision apparatus 200 willbe explained. The information provision apparatus 200 is, for example,an information processing apparatus such as an ASP (Application ServiceProvider) server, and provides the AR service to the information displayapparatus 100 connected through the communication network NW in thisembodiment. The configuration of such information provision apparatus200 will be described with reference to FIG. 5. FIG. 5 is a blockdiagram showing the configuration of the information provision apparatus200.

As shown in the Figure, the information provision apparatus 200 has acontroller 210, a communicator 220, an inputter-outputter 230, a storer240, and/or the like.

The controller 210 has a CPU, a RAM, and/or the like. The controller 210controls each section of the information provision apparatus 200 andperforms each process described later by executing a program.

The communicator 220 is a component for communicating through thecommunication network NW. For example, the communicator 220 isconfigured by a device for communication such as a NIC (NetworkInterface Card), a router, a modem and/or the like, and performs datatransmission and reception using a suitable communication protocol.

The inputter-outputter 230 is configured by an input device such as akeyboard and a mouse, which performs an input according to operation bythe operator of the information provision apparatus 200, and an outputdevice such as a display device, which outputs process results.

The storer 240 is configured by a storage device such as a hard diskdrive, and/or the like. The storer 240 stores the program which thecontroller 210 executes. In addition, the storer 240 stores datarequired for each process performed by execution of the program and datagenerated by the processes.

Functions which are realized by executing programs stored in the storer240 by the controller 210 will be described with reference to FIG. 6.FIG. 6 is a functional block diagram showing functions which arerealized by the controller 210.

In this embodiment, as shown in the Figure, functions of thecommunication processor 211, the information registrator 212, theinformation retriever 213, and/or the like, are realized by thecontroller 210.

The communication processor 211 controls the communicator 220 and,communicates with the information display apparatus 100 through thecommunication network NW. In this embodiment, the communicationprocessor 211 receives the key information transmitted from theinformation display apparatus 100 and, transmits the informationretrieved by the information retriever 213 mentioned later to theinformation display apparatus 100.

The information registrator 212 performs a process for registeringinformation required for providing the AR service into the storer 240.FIG. 7 is an example of information registered in the storer 240. Inthis embodiment, a “reference image storing table” shown in the Figureis generated in the storer 240 and, various kinds of informationrequired for providing the AR service are stored therein.

In the reference image storing table, information (hereinafter, referredto as “index information”) which is index of each reference image,information (hereinafter, referred to as “superimposing information”)for being superimposed and displayed on the captured image correspondingto the subject specified by the reference image, and/or the like, arestored in correspondence with a plurality of reference images preparedin order to apply the matching with the subject taken in the capturedimage which the information display apparatus 100 using the AR servicehas image.

The “reference image” in this embodiment is an image in which the objectrelating to the information which is superimposed on the captured imageby the AR service is captured as the subject. For example, an imagehaving the exterior of some building such as a store as the subject oran image having a product such as commercial product as a subject isregistered as the reference image.

The index information data caused to correspond with such referenceimage is position information (latitude and longitude information),and/or the like, which shows a position where the subject concernedexists, for example. These pieces of information may be obtained asadditional information of Exif (Exchangeable Image File Format), and/orthe like, by applying the captured image taken by the imaging deviceprovided with the GPS function as the reference image, for example. Inaddition, these pieces of information may be input by the operation inwhich the operator of the information provision apparatus 200 operatesthe inputter-outputter 230, when registering into the storer 240.

Other than the index information, “scale information” and “unit scale”as shown in FIG. 7 are recorded in the reference image storing table.

The scale information is to be used for specifying a scale of thesubject among shot data concerning the reference image. An image size(the number of pixels) of the image concerned, a size of imagingelements (sensor size), distance to the subject, a focal length of alens, and/or the like, are recorded as the scale information. Thesepieces of information may be recorded by being acquired from theadditional information of the Exif data relating to the reference image,and/or the like, or by an input by the operator's operation, and/or thelike.

Moreover, the scale information also includes information which shows anangle of view. The angle of view is calculated from the above mentionedscale information and is recorded. In this case, for example, the angleof view [degree] is obtained by calculating the mathematical formula 1.2×tan⁻¹{sensor size [mm]/(2×focal length [mm])}  Mathematical Formula 1

The size of imaging elements is the size of either length or widththereof and, the focal length is a value converted into the lens focalpoint distance in a 35-mm still camera, for example.

The unit scale is information which is calculated based on theabove-mentioned scale information and which shows the scale of thesubject. In this embodiment, for example, an actual size correspondingto 1 pixel (picture element) is recorded as the unit scale. Calculationof such unit scale is performed by the information registrator 212 atthe time of registration of the reference image. Instead, the valuecalculated in advance may be input by the operator.

The calculation of the unit scale is performed according to thefollowing procedures, for example. First, the size of the subject in thereference image is obtained by calculating the mathematical formula 2from the distance to the subject, the size of the image sensor, and thefocal length of the lens.(distance to the subject [mm]×sensor size [mm])/focallength[mm]  Mathematical Formula 2

The size of imaging elements is the size of either length or widththereof and, the focal length is a value converted into the lens focalpoint distance in a 35-mm still camera, for example. According to theabove mentioned calculation, the actual size [mm] of the subject in adirection (length or width) of the substituted image elements size isspecified.

A size in real scale corresponding to 1 pixel is calculated by dividingthe subject size calculated in this manner by the image size (the numberof pixels) in the direction (length or width) of the imaging size usedfor the calculation.

The information retriever 213 retrieves and acquires the correspondingrecord from the reference image storing table based on the keyinformation received from the information display apparatus 100 by thecommunication processor 211. The position information which shows thecurrent position of the information display apparatus 100, and/or thelike, is acquired as the key information, and the record of which theindex information is the acquired key information is specified. Thecommunication processor 211 transmits at least the capturedata of thereference image, the superimposing information, the scale information,the unit scale, and/or the like (hereinafter, referred to as “AR data”),among the information recorded in the specified record, to theinformation display apparatus 100.

Although each of functions described above are logically realized by thecontroller 210 in this embodiment, a part of or all of the functions maybe realized by hardware such as ASIC, for example.

The example of operations by the information display apparatus 100 andthe information provision apparatus 200 having configurations asexplained above will be described below.

First, operations of the information display apparatus 100 will bedescribed with reference to a flowchart shown in FIG. 8. FIG. 8 is theflowchart for explaining the “AR display process (1)” which is performedin the information display apparatus 100 of this embodiment.

When the AR application is started by the AR processor 111, the ARprocessor 111 notifies the fact to the imaging processor 112 and thematching processor 113. The imaging processor 112 which received thenotification controls the imager 140, and an imaging operation isstarted (step S1101).

When the imaging operation is started, the matching processor 113controls the key information acquirer 130, and acquisition operation ofkey information is performed. The acquisition of position information isat least performed by the GPS module, and/or the like. When theacquisition of the position information is the successful, an attempt toacquire other key information is made. When the acquisition of theposition information is unsuccessful, the matching processor 113 waitsuntil acquisition is possible or performs an error process such ascompleting the present process, since the AR service cannot be provided.

Then, when the key information which includes at least positioninformation has been acquired, the matching processor 113 records theacquired key information in the storer 160 (step S1102). The keyinformation recorded here will be deleted at the time of finishing thisprocess.

The matching processor 113 determines whether or not the recorded keyinformation has changed (step S1103). Since there is no last record in acase where the key information is acquired right after starting theprocess, the matching processor 113 determines “changed” (step S1103:Yes).

In this case, the matching processor 113 controls the communicator 120to transmit the acquired key information to the information provisionapparatus 200 through the communication network NW (step S1104).

When the information provision apparatus 200 has received the keyinformation transmitted from the information display apparatus 100 inthis manner, “AR information provision process (1)” for transmitting theAR data to the information display apparatus 100 is performed in theinformation provision apparatus 200. The “AR information provisionprocess (1)” performed in the information provision apparatus 200 ofthis embodiment will be described with reference to a flowchart shown inFIG. 9.

When the communication processor 211 has received the key informationwhich the information display apparatus 100 has transmitted, theinformation retriever 213 acquires the information corresponding to theindex information in the reference image storing table from the receivedkey information (step S1201). The position information which shows ashooting location by the information display apparatus 100 is acquiredfrom the key information.

The information retriever 213 retrieves a record in which the positioninformation corresponding to the shooting location is recorded with theacquired index information (position information) as a retrievingsubject from the reference image storing table in the storer 240 (stepS1202) and, determines whether or not any record in which the positioninformation corresponding to the shooting location is recorded exists inthe reference image storing table (step S1203). In retrieving based onthe position information, the information retriever 213 retrieves notonly the record in which the position information showing the shootinglocation is recorded, but also the record in which the positioninformation showing an area near the shooting location (an area apredetermined distance away from the shooting location) is recorded.

In a case where the record in which the position informationcorresponding to the shooting location by the information displayapparatus 100 is recorded exists in the reference image storing table(step S1203: Yes), the information retriever 213 acquires the AR datafrom the record concerned (step S1204), and transfers the AR data to thecommunication processor 211.

The communication processor 211 controls the communicator 220 totransmit the AR data, which is acquired by the information retriever213, to the information display apparatus 100 through the communicationnetwork NW (step S1205), and finishes the process.

On the other hand, in a case where the record in which the positioninformation corresponding to the shooting location does not exist in thereference image storing table (step S1203: No), the informationprovision apparatus 200 finishes the process without transmitting the ARdata to the information display apparatus 100. In this case, theinformation provision apparatus 200 may transmit notificationinformation showing that the AR service cannot be provided at thecurrent position, and/or the like, to the information display apparatus100.

The transmission and reception operations with the information displayapparatus 100 described above is realized by utilizing a knowntransaction technique which uses CGI (Common Gateway Interface)technique, and/or the like.

Return to the flowchart of FIG. 8, operations of the information displayapparatus 100 to which the AR data is returned will be described.

When the AR data has been transmitted from the information provisionapparatus 200, the communicator 120 of the information display apparatus100 receives the AR data through the communication network NW. When suchreception has been made (step S1105: Yes), the matching processor 113stores the received AR data in the storer 160, and notifies that to theAR processor 111.

The AR processor 111 displays or outputs the live view image which theimager 140 is imaging now by controlling the inputter-outputter 150,generates an image showing distribution of the received reference imagebased on the unit scale of the received AR data, and displays or outputsthe generated image to the inputter-outputter 150 (step S1106).

The contents of the processes will be explained with reference to thedrawings based on the examples of scenes which may be supposed. In thisembodiment, the AR service is utilized in a town with the stores A to Eas shown in FIG. 10A. With respect to the store A, the reference image,and/or the like, is registered in the information provision apparatus200 and the store A can provide the AR service. However, with respect tothe stores B to E, the information is not registered in the informationprovision apparatus 200. Thus the stores B to E can not provide the ARservice.

With respect to the store A, the reference images as shown in FIG. 10Bare registered in the information provision apparatus 200. As shown inthe Figure, a reference image AA which is photographed the externalappearance of the store A and, reference images AB to AZ which arephotographed products available for sale in the store A are registeredin the information provision apparatus 200. All of the image sizes ofthese reference images are the same.

Moreover, the reference images are photographed so that only the targetsubject may be fit within the image size thereof. In such a case where aplurality of reference images about a plurality of subjects areprepared, since the sizes of the subjects are not uniform, the scales ofthe subjects for the images also differ. Specifically, an externalappearance of a building is photographed in full angle of view in thereference image AA whereas a pen is photographed in full angle of viewin the reference images AZ, thus sizes of them in real scalecorresponding to one pixel is quite different.

In this case, if the external appearance of the store A is photographedin full angle of view of the photographed image, the matching with thereference image AA can be performed. However, even if the pen of productis in the image, the matching with the reference image AZ cannot beperformed since the scale of the pen in the image is too small.Specifically, if the photograph is not taken by a suitable angle of viewaccording to the target subject, the information provided by the ARservice cannot be acquired appropriately.

In order to resolve such inconvenience, according to this embodiment,the unit scale registered with the reference image in the referenceimage storing table of the information provision apparatus 200 is usedto guide the user so that the user can shoot a photograph with asuitable angle of view.

Therefore, in this embodiment, the distribution display based on theunit scale of the reference image received from the informationprovision apparatus 200 is performed in the step S1106. A displayexample in this case is shown in FIG. 11A. As shown in the Figure, a“captured image display region” and a “guide display region” are set tothe inputter-outputter 150, and the live view image by the imager 140 isdisplayed on the captured image display region. In the guide displayregion, the distribution display as shown in FIG. 11A is applied.

In this distribution display, as shown in the Figure, for example,arrows which point upwards and downwards along the vertical direction isdisplayed, and on the right side of the upward and downward pointingarrows, the distribution of the reference images based on the unitscales is displayed in a graph manner. In the example of FIG. 11A, thefurther the position of the distribution is upwards along the verticalline with the upward and downward pointing arrows, the farther thedistance from the information display apparatus 100 (camera) to thesubject of the reference image corresponding to the distribution, andthe further the position of the distribution is downwards along thevertical line with the upward and downward pointing arrows, the closerthe distance from the information display apparatus 100 (camera) to thesubject of the reference image corresponding to the distribution.

Specifically, since the unit scale is the size in real scalecorresponding to 1 pixel, the value of the unit scale becomescomparatively large in the image in which the external appearance of thebuilding is shot as the reference image AA (FIG. 10B), and the value ofthe unit scale becomes comparatively small in the image in which a penis shot as the reference image AZ. Accordingly, it means that thesubject is shot from separated place as the value of the unit scale islarge and, it means that the subject is shot from approached place asthe value of the unit scale is small.

For this reason, in the distribution display shown in FIG. 11A,distribution is displayed on upper side of the upward and downwardpointing arrows that shows farther from the subject of the referenceimage as the value of the unit scale becomes large, and distribution isdisplayed on the lower side of the upward and downward pointing arrowswhich shows closer to the subject of the reference image as the value ofthe unit scale becomes small. The horizontal direction of thedistribution display indicates the number of images of the unit scaleconcerned. In the example of FIG. 11A, the distribution displayed nearthe upper right side of the vertical line with the upward and downwardpointing arrows corresponds to the reference image AA (FIG. 10A) inwhich the external appearance of the store is shot, and a plurality ofdistributions near the lower right side of the vertical line with theupward and downward pointing arrows correspond to a plurality ofreference images AB-AZ (FIG. 10B) in which the product is shot.

Returning to the flowchart of FIG. 8, the explanation of the processesis continued. When the distribution display is performed at the stepS1106 as above-described manner, the AR processor 111 calculates theunit scale about the image which have captured by the imager 140 (stepS1107). The unit scale is calculated by a similar manner as theinformation registrator 212 of the information provision apparatus 200described above using the performance information (sensor size, and/orthe like) of the imager 140, the photographing settings at that time(image size, lens focal point distance, and/or the like), the distanceto the subject acquired by the ranging operation. If the imager 140 is astereo camera, since the ranging using the principle of triangulationcan be performed, the distance to the subject can be obtained morecorrectly.

When the unit scale about the captured image has been calculated, the ARprocessor 111 displays the guide as shown in FIG. 11B on the guidedisplay region (step S1108). This guide reflects the calculated unitscale of the captured image and, the guide is displayed on the left sideof the upward and downward pointing arrows in the distribution display.

The live view image, displayed on the captured image display region inFIG. 11B, has a wide angle of view in which not only the store A thatcan provide the AR service but also the stores B to E that can notprovide the AR service are captured. In such case, even if the store Ais included in the angle of view, since the size of the store A in theentire image is small, the scale of the live view image is differentfrom the scale of the reference image AA (FIG. 10B) and the matching maynot be performed appropriately.

In this case, as shown in FIG. 11B, the guide reflecting the unit scaleof the captured image is displayed so as to be located upper than thedistribution showing the reference image AA. In the example of FIG. 11B,since the distribution component of the reference image does not existat a guide position, the user can recognize that the matching forapplying the AR display cannot be performed unless the user approachesthe subject or causes the angle of view to be telescopic side by zoom.

In this way, in a case where it is not possible to apply the matchingusing the reference image acquired from the information provisionapparatus 200 with current captured image (step S1109: No), the processreturns to the step S1107 and the AR processor 111 calculates the unitscale of the captured image again. Specifically, when the user who hasrecognized that the matching cannot be applied according to the guidedisplay causes the angle of view to be narrower as shown in FIG. 12A byapproaching the subject or causing the zoom to be telescopic side, theunit scale for the captured image is calculated (step S1107).

In this case, the guide display is made according to the calculated unitscale (step S1108), and the guide is displayed so as to move downward asshown in FIG. 12A. When the guide moves to a position corresponding todistribution component of the reference image AA as shown in the Figure,it is possible to apply the matching with the reference image AA by thepresent angle of view (step S1109: Yes).

In this case, the matching processor 113 applies the matching thereference image (in this case, the reference image AA) of the unit scalecorresponding to the guide with the captured image, by performing knownimage processing in the AR display (step S1110), and recognizes theposition of the store A in the captured image. When the matching isapplied in such manner, the captured image is displayed in full-screenby the AR processor 111, as well as the superimposing information whichis associated with the reference image AA and provided from theinformation provision apparatus 200 is displayed superimposing on aposition corresponding to the store A in the captured image as shown inFIG. 12B for example (step S1111).

Then, for example, if predetermined termination events, such astermination instruction of the AR application and power off of theinformation display apparatus 100, do not occur (step S1112: No), theprocesses after the step S1102 are performed repeatedly. Although theacquisition operation of the key information is performed in the stepS1102, if the information display apparatus 100 does not move greatly,the key information acquired last time does not change (step S1103: No).In such case, the process proceeds to the step S1107 and the guidedisplay (update of the guide position according to the change of angleof view) based on the current unit scale is performed.

On the other hand, in a case where the key information which is newlyacquired is changed (step S1103: Yes), the key information concerned isanew transmitted to the information provision apparatus 200 (stepS1104), and following processes are performed using the AR data providedbased on the key information (from step S1105).

When the reference image corresponding to the key informationtransmitted from the information display apparatus 100 does not exist inthe information provision apparatus 200, as described above, thenotification information which indicates that the AR data is nottransmitted or the AR service cannot be provided is transmitted from theinformation provision apparatus 200. In such case (step S1105: No), theprocess proceeds to the step S1112, and the operation in accordance withthe existence or absence of the termination event is performed.Specifically, if the termination event does not occur (step S1112: No),processes are repeated again from the acquisition operation of the keyinformation (from step S1102).

In a case where the termination event occurs (step S1112: Yes), the keyinformation recorded in the storer 160 is eliminated, and the processesare finished.

As explained above, according to this embodiment, when the terminaldevice applies the AR display, the distribution based on the subjectscale of the reference image prepared relating to the shooting locationis displayed and, the scale of the subject in the captured image isdisplayed with guide on the distribution display. Thus, it is possibleto easily set the angle of view and shooting location that is suitablefor the matching using the provided reference image and, it is possibleto improve the convenience at the time of utilizing the AR display.

Although the guide displays as shown in FIGS. 11A, 11B and FIGS. 12A,12B are exemplified in this embodiment, the method and configuration ofthe guide displays are arbitrary. Accordingly, the present disclosureshould not be restricted to the present examples. For example, in a casewhere the image of subject in the captured image is small because ofbeing separated from the subject, the information display apparatus 100may display an upward pointing arrow as the guide, as shown in FIG. 13A,for inducing a change of the angle of view by approaching the subject orzoom-in.

Conversely, in a case where the image of subject is not within the angleof view because of being close to the subject too much, the informationdisplay apparatus 100 may display an downward pointing arrow as theguide, as shown in FIG. 13B, for inducing a change of the angle of viewby separating from the subject or zoom-out.

Alternatively, the information display apparatus 100 may display arectangle frame shown in FIG. 13C showing in the angle of view requiredfor the matching as a guide, and induce the change of the angle of viewso that the image of the subject image is to be framed in the rectangleframe.

These guide displays are effective when causing the angle of view to beclose to more suitable angle, for example, in a case where the unitscales of the captured image and the reference image are near. Besides,it is possible to induce an effective change of the angle of view byproviding a more intuitive guide display. Therefore, the guide displayexemplified in FIGS. 11A and 11B may be applied first, then the guidedisplay may change to the guide display as shown in FIGS. 13A to 13C atthe stage in which the unit scales are close to some extent.Alternatively, in a case where the number of the reference images issmall, the guide display as shown in FIGS. 13A to 13C may be performedfrom the beginning.

In this embodiment, although more accurate distance information isacquired by acquiring the distance to the subject which is required tocalculate the unit scale according to the ranging by the stereo camera,the method for acquiring the distance to the subject is arbitrary. Forexample, a focus distance of a lens may be used for acquiring thedistance to the subject. In this case, the imager 140 provided in theinformation display apparatus 100 may not be the exemplified stereocamera, and the imager 140 may be configured by a general monocular-typecamera module.

Second Embodiment

In the above-described first embodiment, when the information displayapparatus 100 has acquired the key information such as positioninformation by GPS, and/or the like, the information provision apparatus200 searches the AR data, based on the key information, to provide theAR data to the information display apparatus 100. However, when theproviders of the AR service are expanded, the providers of the ARservice in the vicinity of the current position are increased. Thus, thenumber of times of inquiry to the information provision apparatus 200 isincreased, the amount of the AR data provided by one time is alsoincreased and, the traffic load or the memory usage in the informationdisplay apparatus 100 is increased, thus there is a possibility ofmaking the performance worse.

Moreover, as mentioned above, the information display apparatus 100 mayacquire two or more kinds of key information items, thus in some cases,a large amount of reference images corresponding to a plurality of keyinformation items may be compared one after the other. This also becomesa factor which makes the performance of the information displayapparatus 100 worse.

Moreover, if the same kind of key information is used, a user may returnto the course before arriving at a target place when using the positioninformation by the GPS, for example. If the information displayapparatus 100 accesses the information provision apparatus 200 inaccordance with a change of the position information, the performance ofthe information display apparatus 100 is decreased and it is notpreferable for a user's convenience.

It is possible to significantly reduce the number of accesses to theinformation provision apparatus 200 if the AR data is stored in theinformation display apparatus 100. However, it takes time for matchingprocess when AR data is stored in the information display apparatus 100,in which the matching is performed. Therefore, a decrease in theperformance is not prevented.

In this embodiment, it is intended to reduce the number of times ofinquiry to the information provision apparatus 200 and to increase anefficiency of the memory usage in the information display apparatus 100by management of the acquired key information and AR data.

The “AR display process (2)” performed by the information displayapparatus 100 in order to achieve this matters will be explained withreference to a flowchart of FIG. 14. This process is executed instead ofthe “AR display process (1)” of the first embodiment.

Whereas the position information acquired by GPS, and/or the like, isnecessary as the key information in the above mentioned firstembodiment, in this embodiment, even if other kind of key information isacquired without acquiring the position information, the informationdisplay apparatus 100 inquires to the information provision apparatus200.

As described above, the key information may be transmitted fromfacilities such as the store. If a transmitting device according to thea short range wireless communication or the visible light communicationsis installed in the front of the store and/or the like, and the keyinformation concerning the AR data about the store is transmitted, thekey information can be acquired at the time of being close to the store.In this case, the key information transmitted can be the positioninformation corresponding to the position information acquired by GPS,and/or the like. In addition, in a case where the AR data, and/or thelike, about the facility is registered in the information provisionapparatus 200, identification information which shows the facility maybe sent as the key information.

In such case, a “group ID” as shown in FIG. 15 is added in the referenceimage storing table of the information provision apparatus 200 as indexinformation. Associating a plurality of AR data to one group ID allowsthe efficiency of the search in the information provision apparatus 200to increase.

Moreover, in this embodiment, unique identification information (AR dataidentification information) is provided to each AR data, as shown inFIG. 15. In this case, the information provision apparatus 200 of thisembodiment transmits the identification information (AR dataidentification information) corresponding to the key information to theinformation display apparatus 100 when the key information istransmitted from the information display apparatus 100 and, theinformation provision apparatus 200 transmits the AR data correspondingto the identification information (AR data identification information)to the information display apparatus 100 when the identificationinformation (the AR data identification information) is transmitted fromthe information display apparatus 100.

The key information acquired by the information display apparatus 100includes one transmitted from an RFID chip attached to a price card,and/or the like, which is provided to the product, and/or the like, soldin the store, and one expressed by a displayed two dimensional code.These key information items are identification information associatedwith the AR data about the products. In this embodiment, suchidentification information is dealt with same one as the AR dataidentification information mentioned above.

The operation according to the “AR display process (2)” performed underthe above assumption will be described. The process starts by anexecution instruction for the AR application by the user as well as thefirst embodiment.

Starting the process, the imaging processor 112 controls the imager 140,and an imaging operation is started (step S2101). Further, the matchingprocessor 113 controls the key information acquirer 130, and acquisitionoperation of key information is performed (step S2102). The acquired keyinformation is temporarily stored on a storage area such as RAM, forexample.

Although the acquired key information is recorded on the storer 160 aswell as the example in the first embodiment, in this embodiment, the“key information management table” as shown in FIG. 18A is created inthe storer 160, and the acquired key information and additionalinformation are stored in this table.

The key information management table is configured by records in which arecord No. is a key as shown in FIG. 18A. The number of recordsconfiguring the table has been defined according to a capacity assignedto the AR application among a capacity of the storer 160 and processingability of the information display apparatus 100, and/or the like.Specifically, the number of records is defined so that the data amountof the key information to be recorded is an amount by which theperformance for execution of the AR application does not decrease. Inthis embodiment, the table is configured by n records, for example.

Moreover, the number of the key information items acquired in the stepS2102 is N (pieces). As described above, since the key information isacquired by several ways, in a case of performing key informationacquisition operation at the same place, a plurality of key informationitems may be acquired simultaneously. For example, the positioninformation is acquired by GPS as well as the group ID is acquired bythe short range wireless communication. Thus, the number of keyinformation items N is N>=1.

The key information management table stores, as shown in FIG. 18A,acquired key information (for example, the position information, thegroup ID, the identification information, and/or the like), informationindicating a type of way by which the key information is acquired (forexample, types such as GPS, visible light communications, RFID, IrDA,Bluetooth (trademark) and two dimensional code, and/or the like), usagedate and time, identification information (AR data identificationinformation) transmitted from the information provision apparatus 200with respect to the key information, a flag indicating that the recordshould be excepted from a deletion candidate (“exclusion flag”), and/orthe like.

The “usage date and time” refers to date and time when the acquired keyinformation acquired is recorded. Moreover, when key informationcorresponding to the key information already recorded on the keyinformation management table is acquired again, the usage date and timeis updated to that time.

A “key information management process” is performed as a process forstoring the key information in such key information management table(step S2200). This “key information management process” will bedescribed with reference to a flowchart shown in FIG. 16.

Starting the process, the matching processor 113 accesses the keyinformation management table in the storer 160 and, in a case where thesame key information as the key information stored in the RAM is storedin the key information management table, updates the “usage date andtime” corresponding to the key information to current date and time(step S2201). The key information that has been completely recorded inthe key information management table among the key information presentlyacquired and stored in the RAM is deleted from the RAM after updatingthe usage date and time.

The key information remaining in the RAM after deleting the recorded keyinformation from the RAM is not recorded on the key informationmanagement table. In a case where the key information which is notrecorded in the key information management table does not exist amongthe presently acquired key information (step S2202: No), a process iscompleted by updating the usage date and time alone, and then returningto the flow of the “AR display process (2)” of FIG. 14.

On the other hand, in a case where the key information which is notrecorded in the key information management table exists among thepresently acquired key information (step S2202: Yes), the matchingprocessor 113 determines whether or not such unrecorded key informationcan be added into the key information management table (step S2203). Asdescribed above, the number of records of the key information managementtable concerning this embodiment is limited by n, based on the capacityof the storer 160, and/or the like. If there are one or more emptyrecords among such n records, the matching processor 113 determines thatthe key information can be added (step S2203: Yes).

In this case, the matching processor 113 records, with respect to thekey information which remains in the RAM, the key information per se,the information which shows acquisition manner, and the informationwhich shows a present date and time as acquisition date and time in oneof the empty records (step S2209). In this case, the matching processor113 deletes the key information on the RAM.

When one of the key information acquired this time has been recorded onthe key information management table, the matching processor 113controls the communicator 120 so as to transmit the key information tothe information provision apparatus 200 through the communicationnetwork NW and, acquires corresponding identification information (theAR data identification information) from the information provisionapparatus 200 through the communication network NW (step S2210).

In the information provision apparatus 200, when the communicator 220has received the key information transmitted from the informationdisplay apparatus 100, the information retriever 213 accesses the storer240 to retrieve the identification information (the AR dataidentification information) associated with the received key informationfrom the reference image storing table shown in FIG. 15. Then thecommunicator 220 transmits the identification information (the AR dataidentification information) to the information display apparatus 100.

When the information provision apparatus 200 transmits theidentification information (the AR data identification information) inthis way, the communicator 120 of the information display apparatus 100receives the identification information (the AR data identificationinformation). The matching processor 113 records the receivedidentification information (the AR data identification information) intothe record in which the key information is recorded in the keyinformation management table in the step S2209 (step S2211).

When recording the key information in one record is completed, thematching processor 113 determines whether or not all of the unrecordedkey information in the key management information table has beenrecorded, by determining whether or not the key information acquiredthis time remains on the RAM (step S2212).

If the key information acquired this time still remains on the RAM (stepS2212: No), the process returns to the step S2203. In a case whereinformation is recorded in all n records, the matching processor 113determines the key information cannot be added (step S2203: No), andprepares an empty record by deleting contents in any record.

In such situation, the matching processor 113 scans the usage date andtime recorded in the key information management table, and specifies therecord in which the oldest usage date and time is recorded (step S2204).Then, the matching processor 113 determines whether or not the keyinformation recorded in the record is ones other than positioninformation (such as group ID and identification information) (stepS2205).

In a case where the key information which is of other than positioninformation is recorded in the record (step S2205: Yes), the matchingprocessor 113 deletes the contents of the record (step S2208). Since oneempty record is prepared in this way, the information is recorded intothe empty record by the processes of the step S2209 and following steps.

On the other hand, in a case where the key information in the record inwhich the oldest usage date and time is recorded is the positioninformation (step S2205: No), the matching processor 113 determineswhether or not the position information recorded in the record concernedis a position close to a current position, based on the positioninformation and the current position (step S2206).

If position information is included in the key information acquired thistime, the matching processor 113 uses the position which the positioninformation shows as a current position. If position information is notincluded, the matching processor 113 acquires current positioninformation by controlling the key information acquirer 130. Then, thematching processor 113 determines whether or not the position which theposition information shows is close to the current position bycalculating a distance between two points from the latitude andlongitude which show each position thereof and, comparing the calculateddistance with a predetermined threshold. The threshold in this case maybe arbitrary. The threshold may be set up by the manufacturer of theinformation display apparatus 100, and also may be arbitrarily set up bythe user.

If the position information recorded in the record with the oldest usagedate and time shows the position close to the current position accordingto such operation (step S2206: Yes), the matching processor 113 excludesthe record concerned from a deletion candidate (step S2207), and returnsto the step S2204. In this case, a flag (exclusion flag) showing thatthe record should be excluded from the deletion candidate is set in therecord, for example.

On the other hand, if the record is that the position informationshowing a position far from the current position is recorded therein(step S2206: No), the process proceeds to a step S2208 and the matchingprocessor 113 deletes the contents in the record. Then, each kind ofinformation items is recorded in the record concerned by performing theprocesses of the step S2209 and following steps.

Thus, when all of the key information acquired this time has beenrecorded in the key information management table (step S2212: Yes), thematching processor 113 finishes this process and returns to the flow ofthe “AR display process (2)” (FIG. 14).

In this embodiment, the AR data acquired from the information provisionapparatus 200 is recorded in an “AR data management table” as shown inFIG. 18B generated in the storer 160.

As shown in the Figure, records in which record No. is used as a key arecreated in the “AR data management table”. Moreover, in each record, theidentification information (the AR data identification information)corresponding to the key information is recorded. In addition, AR data(particularly, the reference image, the superimposing information, thescale information, the unit scale information, and/or the like)corresponding to the identification information (the AR dataidentification information), and information showing the usage date andtime of the AR data, and/or the like, are recorded. The date and timewhen the AR data is acquired from the information provision apparatus200 is recorded as the “usage date and time.”

The number of records configuring the AR data management table has alsobeen defined according to the capacity of the storer 160, the capacitybeing assigned to the AR application, and processing ability of theinformation display apparatus 100. Specifically, the number of recordsis defined so that the amount of information to be recorded is an amountby which the performance for execution of the AR application does notdecrease. In this embodiment, the table is configured by m records.

In the “AR display process (2)”, after executing the above-described“key information management process” (step S2200), an “AR datamanagement process” for recording the AR data in such AR data managementtable is performed (step S2300). This AR data management process will beexplained with reference to a flowchart shown in FIG. 17.

Starting the process, first, the matching processor 113 accesses the keyinformation management table (FIG. 18A) in the storer 160, specifies therecord in which the date and time corresponding to the key informationacquisition operation of this time is recorded as the usage date andtime and, specifies the identification information (the AR dataidentification information) recorded in the record. Specifically, thematching processor 113 specifies the identification information (the ARdata identification information) associated with the key informationwhich is acquired this time. In this case, the matching processor 113acquires the specified identification information (the AR dataidentification information) from the key information management table(step S2301), and stores the acquired identification information (the ARdata identification information) temporarily in the RAM.

Next, the matching processor 113 accesses the AR data management table(FIG. 18B) in the storer 160, specifies the record in which theidentification information (the AR data identification information)acquired in the step S2301 has already recorded and, updates the “usagedate and time” of the record concerned to current date and time (stepS2302). Specifically, in a case where the AR data, associated with thekey information which is acquired this time, has already been recordedin the AR data management table, the “usage date and time” is updated asit is considered that the AR data is acquired from the informationprovision apparatus 200 at this time.

When the “usage date and time” has been updated with respect to theidentification information (the AR data identification information)which shows that the AR data has been recorded in the AR data managementtable, the identification information is deleted from the RAM. If theidentification information remains in the RAM after such deletion, theAR data shown by the identification information remaining in the RAM isnot recorded in the AR data management table.

Moreover, if the identification information does not remain in the RAM,all of the AR data corresponding to the key information acquired thistime has been recorded in the AR data management table. In this case(step S2303:No), the matching processor 113 finishes the process afteronly updating the “usage date and time” with respect to the AR datacorresponding to the key information acquired this time and, the processreturns to the flow of the “AR display process (2)” (FIG. 14).

On the other hand, in a case where the AR data which is not recorded inAR data management table exists (step S2303: Yes), the matchingprocessor 113 determines whether or not the AR data can be added intothe AR data management table (step S2304). As described above, thenumber of records of the AR data management table concerning thisembodiment is limited by m based on the capacity of the storer 160,and/or the like. If one or more empty records exist among such mrecords, the matching processor 113 determines that the AR data can beadded (step S2304: Yes).

In this case, the matching processor 113 selects one of theidentification information (the AR data identification information)stored in the RAM. Then, the matching processor 113 controls thecommunicator 120 to transmit the selected identification information(the AR data identification information) to the information provisionapparatus 200 through the communication network NW and, acquires the ARdata associated with the identification information (the AR dataidentification information) from the information provision apparatus 200(step S2310).

In the information provision apparatus 200, when the communicator 220has received the identification information (the AR data identificationinformation) transmitted from the information display apparatus 100, theinformation retriever 213 accesses the storer 240 to retrieve the ARdata associated with the received identification information (the ARdata identification information) from the reference image storing tableshown in FIG. 15. Then the communicator 220 transmits the AR data to theinformation display apparatus 100.

When the information provision apparatus 200 transmits the AR data inthis way, the communicator 120 of the information display apparatus 100receives the AR data. The matching processor 113 records the received ARdata in one of the empty records (step S2311). The matching processor113 also records the information which shows the date and time of thistime as the usage date and time of the AR data. At this time, thematching processor 113 deletes the identification information (the ARdata identification information) relating to the AR data recorded in theAR data management table from the RAM, among the identificationinformation (the AR data identification information) stored in the RAM.

The matching processor 113 determines whether or not all of the AR datathat had not been recorded in the AR data management table has beenrecorded in the AR data management table, by determining whether or notthe identification information (the AR data identification information)remains on the RAM (step S2312).

In a case where the identification information (the AR dataidentification information) remains on the RAM (step S2312: No), theprocess returns to the step S2304 and, the processes of acquisition andrecord of the AR data are performed with respect to the identificationinformation (the AR data identification information). In a case whereinformation is recorded in all m records of the AR data managementtable, the matching processor 113 determines that the AR data cannot beadded (step S2304: No) and, prepares an empty record by deleting thecontents in any record.

In such situation, the matching processor 113 scans the usage date andtime recorded in the AR data management table, and specifies the recordin which the oldest usage date and time is recorded (step S2305). Then,the matching processor 113 specifies the identification information (ARdata identification information) currently recorded on the record.

Next, the matching processor 113 accesses the key information managementtable to determine whether a record in which the specifiedidentification information (the AR data identification information) isrecorded exists. Specifically, the matching processor 113 determineswhether or not the key information corresponding to the AR data with theoldest usage date and time remains in the present key informationmanagement table (step S2306).

In a case where the corresponding key information does not remain in thekey information management table (step S2306: No), the matchingprocessor 113 deletes the contents in the record specified in the stepS2305 from the AR data management table (step S2307). Since one emptyrecord is prepared in this way, the information corresponding to therecord is recorded by the processes after the step S2310.

On the other hand, in a case where the key information corresponding tothe AR data with the oldest usage date and time remains in the presentkey information management table (step S2306: Yes), the matchingprocessor 113 determines whether or not the key information has beenexcluded from the deletion candidate in the step S2207 of the “keyinformation management process” (FIG. 16), by determining whether or notthe “exclusion flag” is set in the record relating to the keyinformation (step S2308).

In a case where the key information corresponding to the AR dataspecified in the step S2305 has been excluded from the deletioncandidate in the key information management table (step S2308: Yes), thematching processor 113 also excludes the record specified in the stepS2305 from the deletion candidate (step S2309), and the process returnsto the step S2305. In this case, for example, the record specified inthe step S2305 may be excluded by a flag which notifies object exclusionin the record, or updating the usage date and time in the record to thepresent date and time.

According to the “key information management process” (FIG. 16)described above, the key information management table is managed so thatthe key information acquired comparatively recently or the keyinformation of which the usage date and time is old but which shows aposition close to the current position may remain. Therefore, if the keyinformation corresponding to the AR data specified in the step S2305remains in the key information management table and the key informationhas been excluded from the deletion candidate, there is a highpossibility that the AR data would be utilized for the AR display usingthe key information acquired this time or the AR display using the keyinformation which would be acquired in the near future, even if theusage date and time of the AR data is old. Therefore, such AR data isexcluded from the deletion candidate from the AR data management table,and is made to remain in the AR management table.

The process returns to the step S2305 after such exclusion process, arecord which is the deletion candidate is retrieved and the contentstherein are deleted (the step S2305 to the step S2307). Then, the ARdata newly acquired is recorded into the empty record prepared in thisway (the step S2310 to the step S2311).

On the other hand, even if corresponding key information remains in thekey information management table, if the key information has not beenexcluded from the deletion candidate (step S2308: No), the usage dateand time of the key information is relatively old and a position whichthe key information shows is not relatively close to the currentposition among those which remain in the key information managementtable, and the usage date and time is the oldest with respect to thecorresponding AR data. Accordingly, the matching processor 113 considersthat there is low possibility that the AR data currently recorded in therecord specified in the step S2305 will be used this time or the nearfuture, and the matching processor 113 deletes the contents of therecord concerned to prepare an empty record (step S2307).

Operations as described above are repeated until all the targeted ARdata is recorded (step S2312: No). When all the AR data has beenrecorded (step S2312: Yes), the matching processor 113 finishes thisprocess and the process returns to the flow of the “AR display process(2)” (FIG. 14).

In the “AR display process (2)”, the AR processor 111 applies the ARdisplay using the AR data corresponding to the key information which isacquired in the step S2102. The AR display using acquired AR data isperformed by performing the similar operations with the step S1106 tothe step S1111 in the “AR display process (1)” (FIG. 8), which has beendescribed in the first embodiment (step S2103).

Then, if a predetermined termination event does not occur (step S2104:No), the AR processor 111 applies the AR display by repeatedly executingthe processes after the step S2102. Occurring the termination event(step S2104: Yes) causes this process to finish.

As described above, according to this embodiment, the key informationand the AR data which are stored in the information display apparatus100 are appropriately managed so that the AR display can be performedefficiently, according to a memory resource and processing ability,and/or the like, which can be utilized by the information displayapparatus 100. Furthermore, the data transfer amount for providing theAR data from the information provision apparatus 200 to the informationdisplay apparatus 100 can be reduced effectively. Therefore, it ispossible to reduce traffic load and to prevent the performance fall ofthe information display apparatus 100, thus it is possible to improvethe convenience concerning the use of the AR service.

Specifically, since only the identification information corresponding tothe key information is acquired from the information provision apparatus200 when the key information has been acquired in the informationdisplay apparatus 100, AR data with a comparatively large size, such asthe reference image, is not transmitted. Then, only when the AR datacorresponding to the AR data stored in the information display apparatus100 is not included based on the acquired identification information,the information display apparatus 100 receives a provision of the ARdata from the information provision apparatus 200. Thereby, it ispossible to significantly reduce the data transfer amount on thecommunication network NW, in comparison with the case of acquiring theAR data from the information provision apparatus 200 for each time ofacquiring the key information.

Moreover, the key information acquired comparatively recently and thekey information which shows the position information close to thecurrent position are stored in the information display apparatus 100 soas to remain preferentially and, the AR data corresponding to thesepieces of key information is managed so as not to be deleted. Thus, theAR data with high possibility for being utilized at the present or thelatest future is preferentially stored in the information displayapparatus 100. Therefore, it is possible to further reduce the number oftransaction with the information provision apparatus 200 for acquiringthe AR data.

Although the “usage date and time” about the AR data is updated in thisembodiment, in a case where the AR data corresponding to the keyinformation acquired at that time has already recorded in the AR datamanagement table (Step S2302 in the “AR data management process” (FIG.17)), the “usage date and time” may also be updated in a case of usingthe AR data recorded in the AR data management table for the AR display.Thereby, it is possible to preferentially store the AR data actuallyused for the AR display in the information display apparatus 100.

Moreover, when the key information is deleted from the key informationmanagement table (Step S2208 in the “key information management process”(FIG. 16)), the AR data associated with the key information may bedeleted from the AR data management table. Further, when the AR data isdeleted from the AR data management table (Step S2307 in the “AR datamanagement process” (FIG. 17)), the key information, with which theidentification information of the AR data (the AR data identificationinformation) is associated, may be deleted from the key informationmanagement table. Accordingly, when either key information or AR data isdetermined as a deletion candidate, the AR data or the key informationcorresponding thereto is also deleted. Thus, it is possible to improve apossibility that the empty record is prepared in advance in both of thekey information management table and the AR data management table.

Third Embodiment

Although, a group ID is exemplified as the key information which is sentfrom the facility such as the store, and/or the like, in theabove-described second embodiment, the AR data (the reference image) canbe acquired other than the key information from the facility in the samemanner as the acquired group ID. In this case, for example, even if theposition information cannot be acquired by GPS, the AR data which isrequired for use of the AR service can be acquired. In addition, sincethe AR data can be acquired without accessing the information provisionapparatus 200, and/or the like, the traffic load in the communicationnetwork NW can be reduced.

In this embodiment, a short range wireless communication device andvisible-light-communication device which are installed in the facilitysuch as the store, and/or the like, function as the informationprovision apparatus 200. Different from the information provisionapparatus 200 exemplified in each of above-described embodiments, thesedevices are only necessary to store and send the information regarding ainstallation location. Thus, it is possible to realize on a small scaleand in simple, and to provide a huge amount of AR data dispersedly.

In each of the above-described embodiments, so-called 2D image such as aphotograph image is used as such reference image provided. In this case,unless the image is taken in a direction of line of sight which isidentical to a direction of line of sight from the camera by which thereference image is taken, the above matching may not work well. Inparticular, in the case where AR displays may be applied for a number ofproducts respectively inside a store, and/or the like, if the inside thestore is narrow or the subject is near, a photograph cannot necessarilybe taken in the same direction of line of sight as that of the referenceimage. Moreover, the matching operation is performed in the backgroundand a user does not recognize what kind of reference image is used,thus, usually, it is not possible to take a photograph with a directionof line of sight which is set so as to match with that of the referenceimage.

In a case of the information display apparatus 100 in which the imager140 is configured by the stereo camera, the information displayapparatus 100 can generate 3D modeling data from the captured image.Thus, the information display apparatus 100 attempts to solve theproblems described above, by making such 3D modeling data apply thematching with the reference image. This embodiment will explain anexample of operation in such case below.

An “AR display process (3)” in which the information display apparatus100 applies the matching using 3D modeling data will be explained withreference to a flowchart shown in FIG. 19. In this embodiment, the ARdata is provided to the information display apparatus 100 by the shortrange wireless communication device and the visible-light-communicationdevice which are installed in the facility such as the store.

Starting the process, the imaging processor 112 controls the imager 140which is the stereo camera, and imaging operation is started (stepS3101). In this embodiment, for example, the storer 160 includes astorage apparatus on which capture data is decompressed for imageprocessing, such as VRAM (Video RAM). In this case, the matchingprocessor 113 generates 3D modeling data of the extracted subject byextracting the subject from the captured image and performing a depthestimation, to the capture data decompressed on the storer 160 using aknown technique in the 3D modeling (step S3102).

When 3D modeling data is generated, the matching processor 113 extractscharacteristic points and calculates characteristic amounts, withrespect to the subject for which the 3D modeling data is generated fromthe captured image, using a known technique in the image processing bySIFT (Scale Invariant Feature Transform) or SURF (Speeded Up RobustFeatures), and/or the like (step S3103).

Moreover, the matching processor 113 controls the key informationacquirer 130 or the like to acquire the AR data sent from the shortrange wireless communication device or the visible-light communicationsdevice which are installed in the facility such as the store, andrecords the AR data on the storer 160 (step S3104). With respect to therecord of the AR data, the matching processor 113 records on the AR datamanagement table, and/or the like, as exemplified in the secondembodiment.

In a case where two or more reference images are included in acquired ARdata, in order to apply the matching between generated 3D modeling dataand these reference images one by one, the matching processor 113 setsan index number i of the reference image to be a processing object intoan initial value “0” (step S3105), then applies the value of i+1 (stepS3106). Thus a first reference image is designated.

With respect to the above mentioned reference image (i-th referenceimage) stored in the storer 160. The matching processor 113 performs aprocess similar to that of the step S3103, to thereby extract thecharacteristic points or calculate the characteristic amounts by SIFT,SURF, and/or the like (step S3107).

The subject currently captured is a three-dimensional object as shown inFIG. 20A, whereas the prepared reference image is a photograph image (2Dimage) as shown in FIG. 20B captured from a direction of “line of sightfor reference image” in FIG. 20A. When the current imaging direction ofthe information display apparatus 100 is a direction of “current line ofsight from the camera” in FIG. 20A, the 3D modeling data generated inthe step S3102 would be one as shown in FIG. 20C.

In this case, since the line of sight for the reference image and theline of sight in the current imaging are different, it is not possibleto apply the matching using the reference image for the generated 3Dmodeling data. For this reason, the matching processor 113 comparescharacteristic points, characteristic amounts of the captured image, andcharacteristic points, characteristic amounts of the reference imagerespectively and, changes the direction of line of sight for the 3Dmodeling data so as to correspond to the direction of line of sight forthe reference image (step S3108). The 3D modeling data which is made aline of sight change in this way is similar to the reference image asshown in FIG. 20D. Therefore it is possible to perform the matchingusing the reference image.

In a case where the reference image compared now does not correspond tothe subject for which the 3D modeling data is generated, the matchingdoes not succeed even if the line of sight of the 3D modeling data ischanged (step S3109: No). In this case, the process returns to the stepS3106 and, the matching processor 113 performs above-described processeswith respect to next reference image by making the index number i of thetarget reference image +1.

In a case where the matching has succeeded (step S3109: Yes), the ARprocessor 111 applies the AR display (step S3110) by performing theprocesses similar to the step S1106 to the step S1111 in the “AR displayprocess (1)” (FIG. 8) exemplified in the first embodiment.

Then, if a predetermined termination event which is exemplified in eachof the above-described embodiments does not occur (step S3111: No), theprocess returns to the step S3104, and the matching processor 113repeatedly performs subsequent processes. If the termination eventoccurs (step S3111: Yes), the matching processor 113 finishes thisprocess.

According to this embodiment as described above, since the direction ofthe line of sight for the 3D modeling data generated from the capturedimage is changed based on the characteristic points and thecharacteristic amounts which are extracted from the captured image andthe reference image, it is possible to appropriately apply the matchingeven if the captured image is taken from a direction which is differentfrom the direction of line of sight for the reference image and, it ispossible to adequately apply the AR display regardless of whichdirection the captured image is taken from. Specifically, even in a casewhere the subjects of the AR service according to expansion of the ARservice increase, an exact AR display can be applied and a user'sconvenience can be improved.

Although the reference image as a photograph image is acquired from theshort range wireless communication device or thevisible-light-communications device, which are installed in the facilitysuch as the store, and/or the like, in this embodiment, the referenceimage itself which is provided in this way may be 3D modeling data. Inthe cases where the information provision apparatus 200, exemplified inthe first and second embodiments, provide the reference image, thereference image may be similarly 3D modeling data. When the matching isapplied to those of 3D modeling data, it is possible to perform moreaccurate matching operation based on numerical operation, withoutperforming the image processing.

As explained above, according to each of above-mentioned embodimentsconcerning the present disclosure, it is possible to improve a user'sconvenience even if the number of objects becomes large, in a case ofutilizing the AR service.

The above-mentioned embodiments are examples and, the scope of thepresent disclosure is not restricted to the embodiments. Specifically,various applications are possible and every possible embodiments areincluded in the scope of the present disclosure.

The specifications, the format, and/or the like, which are shown in eachof the above mentioned embodiments are merely examples, and so long asthe same effect is obtained, specifications or format which may beemployed is arbitrary.

Moreover, although the smartphone is exemplified as the informationdisplay apparatus 100 in each of the above-mentioned embodiments, theinformation display apparatus 100 is not restricted to the smartphonebut arbitrary apparatus can be used as the information display apparatus100, if the apparatus has at least an imaging function, and aconfiguration capable of acquiring the key information and/or the ARdata.

In the arbitrary apparatus adoptable as the information displayapparatus 100, even if the apparatus does not have functions realized bythe controller 110 mentioned above, it is possible to cause existingapparatus to function as the information display apparatus concerningthe present disclosure if functional configurations as shown in FIG. 4can be achieved by applying a program.

Similarly, if functional configurations as shown in FIG. 6 can beachieved by applying a program to a general-purpose server device,and/or the like, it is possible to cause existing information processingdevice, and/or the like, to function as the information provisionapparatus concerning the present disclosure.

A method for applying the program to an arbitrary apparatus so as tofunction as the information display apparatus or the informationprovision apparatus may be appropriately determined where necessary. Forexample, the program may be applied to the arbitrary apparatus bystoring the program in the storage media such as CD-ROM and a memorycard, and may be also applied to the arbitrary apparatus through thecommunication media, such as the Internet.

Having described and illustrated the principles of this application byreference to one (or more) preferred embodiment(s), the preferredembodiment(s) of the present disclosure may be readily recognized ascapable of being modified to various configuration without departingfrom the principles disclosed herein and being construed as includingall such modifications and variations within the spirit and scope of thesubject matter disclosed in these embodiments.

What is claimed is:
 1. An information display system for displayinginformation comprising: an information display apparatus which includesan imaging unit; and an information provision apparatus; wherein theinformation display apparatus comprises: a key information acquiringunit which acquires key information corresponding to a subject in acaptured image captured by the imaging unit; and a key informationnotifying unit which notifies the key information to the informationprovision apparatus each time the key information is acquired, whereinthe information provision apparatus comprises: an AR data storage unitwhich stores augmented reality (AR) data corresponding to identificationinformation for identifying the AR data and key information notified bythe key information notifying unit, the AR data including (i) areference image to be compared with the subject in the captured imagecaptured by the imaging unit, and (ii) superimposing information to besuperimposed on the captured image that includes the subject comparedwith the reference image; and an information providing unit (i) whichtransmits the identification information corresponding to the keyinformation to the information display apparatus, without transmittingthe AR data corresponding to the identification information, when thekey information is provided from the information display apparatus, and(ii) which transmits the AR data corresponding to the identificationinformation to the information display apparatus when the identificationinformation is provided from the information display apparatus, whereinthe information display apparatus further comprises: an identificationinformation acquiring unit which acquires the identification informationprovided from the information provision apparatus in accordance with anotification of the key information, without acquiring the AR datacorresponding to the identification information; a key informationrecording unit which adds and records new key information and newidentification information in a table each time the new identificationinformation is acquired, wherein the table contains first sets eachcomprising the key information and the identification information inassociation with each other, wherein the first sets do not include theAR data corresponding to the identification information; anidentification information notifying unit which notifies theidentification information recorded in the table to the informationprovision apparatus; an AR data acquiring unit which acquires the ARdata provided from the information provision apparatus in accordancewith a notification of the identification information; an AR datarecording unit which records second sets in the table other than thefirst sets, the second sets each comprising the identificationinformation notified by the identification information notifying unitand the AR data acquired by the AR data acquiring unit in associationwith each other; and an AR data displaying unit which superimposes anddisplays the superimposing information corresponding to the referenceimage on the captured image based on a result from matching the subjectin the captured image with the reference image included in the AR datawhich is recorded by the AR data recording unit, wherein theidentification information notifying unit determines, each time theidentification information is acquired, whether the AR datacorresponding to the identification information is already recorded inthe table, and (i) if the AR data corresponding to the identificationinformation is determined to already be recorded, the identificationinformation notifying unit does not notify the identificationinformation to the information provision apparatus and the AR dataacquiring unit does not reacquire the AR data which is already recorded,and (ii) if the AR data is determined to not be recorded, theidentification information notifying unit notifies the identificationinformation to the information provision apparatus, the informationproviding unit of the information provision apparatus transmits the ARdata corresponding to the identification information to the informationdisplay apparatus in response to the notification of the identificationinformation by the information display apparatus, the AR data acquiringunit of the information display apparatus acquires the AR datatransmitted from the information provision apparatus, and the AR datarecording unit records the newly acquired AR data in association withthe corresponding identification information as a new second set in thetable.
 2. An information display apparatus comprising: an imaging unit;a communication unit which communicates with an external apparatus; akey information acquiring unit which acquires key informationcorresponding to a subject in a captured image captured by the imagingunit; a key information notifying unit which notifies the keyinformation acquired by the key information acquiring unit to theexternal apparatus through a communication network each time the keyinformation is acquired; an identification information acquiring unitwhich acquires identification information for identifying augmentedreality (AR) data which is to be transmitted by the external apparatus,the identification information acquiring unit acquiring theidentification information without acquiring the AR data correspondingto the identification information, in accordance with a notification ofthe key information by the key information notifying unit through thecommunication network, the AR data including (i) a reference image to becompared with the subject in the captured image captured by the imagingunit, and (ii) superimposing information to be superimposed on thecaptured image that includes the subject compared with the referenceimage; a key information recording unit which adds and records new keyinformation and new identification information in a table each time thenew identification information is acquired, wherein the table containsfirst sets each comprising the key information and the identificationinformation in association with each other, wherein the first sets donot include the AR data corresponding to the identification information;an identification information notifying unit which notifies theidentification information recorded in the table to the externalapparatus; an AR data acquiring unit which acquires the AR datatransmitted by the external apparatus in accordance with a notificationof the identification information; an AR data recording unit whichrecords second sets in the table other than the first sets, the secondsets each comprising the identification information notified by theidentification information notifying unit and the AR data acquired bythe AR data acquiring unit in association with each other; and an ARdata displaying unit which superimposes and displays the superimposinginformation corresponding to the reference image on the captured imagebased on a result from matching the subject in the captured image withthe reference image included in the AR data which is recorded by the ARdata recording unit, wherein the identification information notifyingunit determines, each time the identification information is acquired,whether the AR data corresponding to the identification information isalready recorded in the table, and (i) if the AR data is determined toalready be recorded, the identification information notifying unit doesnot notify the identification information to the external apparatus andthe AR data acquiring unit does not reacquire the AR data which isalready recorded, and (ii) if the AR data is determined to not berecorded, the identification information notifying unit notifies theidentification information to the external apparatus, the AR dataacquiring unit acquires the AR data which is transmitted by the externalapparatus in response to the notification of the identificationinformation, and the AR data recording unit records the newly acquiredAR data in association with the corresponding identification informationas a new second set in the table.
 3. The information display deviceaccording to claim 2, wherein the table comprises: a first table inwhich the first sets each comprising the key information and theidentification information in association with each other are recordedby the key information recording unit, and a second table in which thesecond sets each comprising the identification information and the ARdata in association with each other are recorded by the AR datarecording unit.
 4. The information display device according to claim 3,wherein: the first table and the second table are each defined by anumber of records, the identification information notifying unitnotifies the identification information to the external apparatus whenthe identification information recorded in the first table is notrecorded in the second table; the key information recording unitexecutes a deletion based on a usage date and time of the keyinformation already recorded, when no empty record is present in thefirst table at a time when the key information acquiring unit acquiresthe key information, and does not execute a deletion when an emptyrecord is present; and the AR data recording unit executes a deletionbased on a usage date and time of the AR data already recorded, when noempty record is present in the second table at the time theidentification information notifying unit notifies the identificationinformation, and does not execute a deletion when an empty record ispresent.
 5. The information display apparatus according to claim 4,wherein: the key information recording unit records a date and time whenacquiring the key information as the usage date and time of the keyinformation, and updates the date and time of the key information whensuperimposing and displaying the superimposing information on thecaptured image using the AR data corresponding to the key information;the AR data recording unit records a date and time when acquiring the ARdata as the usage date and time of the AR data, and updates the date andtime when superimposing and displaying the superimposing information onthe captured image using the AR data; and the key information recordingunit and the AR data recording unit specify a record recording an oldestdate and time as the usage date and time as a deletion target.
 6. Theinformation display apparatus according to claim 5, wherein: the keyinformation includes position information indicating a position of theinformation display apparatus at a time of imaging by the imaging unit;and the key information recording unit: (i) calculates a distancebetween the position indicated by the position information and a currentposition of the information display apparatus when the key informationrecorded in the record specified as the deletion target is the positioninformation; and (ii) excludes the record from being the deletion targetwhen the calculated distance is smaller than a predetermined threshold.7. The information display apparatus according to claim 6, wherein theAR data recording unit excludes the specified record from being thedeletion target, when (i) the identification information recorded in therecord specified as the deletion target is recorded in the first table,and (ii) the record recording the identification information in thefirst table has been excluded from being the deletion target by the keyinformation recording unit.
 8. An information display method,comprising: acquiring key information corresponding to a subject in acaptured image captured by an imaging unit; notifying the keyinformation to an external apparatus through a communication networkeach time the key information is acquired; acquiring identificationinformation for identifying augmented reality (AR) data which is to betransmitted by the external apparatus, the identification informationbeing acquired without acquiring the AR data corresponding to theidentification information, in accordance with a notification of the keyinformation through the communication network, the AR data including (i)a reference image to be compared with the subject in the captured imagecaptured by the imaging unit and (ii) superimposing information to besuperimposed on the captured image that includes the subject comparedwith the reference image; adding and recording new key information andnew identification information in a table each time the newidentification information is acquired, wherein the table contains firstsets each comprising the key information and the identificationinformation in association with each other, wherein the first sets donot include the AR data corresponding to the identification information;notifying the identification information recorded in the table to theexternal apparatus; acquiring the AR data transmitted by the externalapparatus in accordance with a notification of the identificationinformation; recording second sets in the table other than the firstsets, the second sets each comprising the notified identificationinformation and the acquired AR data in association with each other;superimposing and displaying the superimposing information correspondingto the reference image on the captured image based on a result frommatching the subject in the captured image with the reference imageincluded in the AR data which is recorded in the table, wherein eachtime the identification information is acquired, it is determinedwhether the AR data corresponding to the identification information isalready recorded in the table, and (i) if the AR data is determined toalready be recorded, the identification information is not notified tothe external apparatus and the AR data which is already recorded is notreacquired, and (ii) if the AR data is determined to not be recorded,the identification information is notified to the external apparatus,the AR data which is transmitted by the external apparatus in responseto the notification of the identification information is acquired, andthe newly acquired AR data is recorded in association with thecorresponding identification information as a new second set in thetable.
 9. A non-transitory computer-readable storage medium which storesa program for causing a computer of an information display apparatusincluding an imaging unit, a communicating unit which communicates withan external apparatus, and a displaying unit, to perform functionscomprising: acquiring key information corresponding to a subject in acaptured image captured by the imaging unit; notifying the keyinformation to the external apparatus through a communication networkeach time the key information is acquired; acquiring identificationinformation for identifying augmented reality (AR) data which is to betransmitted by the external apparatus, the identification informationbeing acquired without acquiring the AR data corresponding to theidentification information, in accordance with a notification of the keyinformation through the communication network, the AR data including (i)a reference image to be compared with the subject in the captured imagecaptured by the imaging unit and (ii) superimposing information to besuperimposed on the captured image that includes the subject comparedwith the reference image; adding and recording new key information andnew identification information in a table each time the newidentification information is acquired, wherein the table contains firstsets each comprising the key information and the identificationinformation in association with each other, wherein the first sets donot include the AR data corresponding to the identification information;notifying the identification information recorded in the table to theexternal apparatus; acquiring the AR data transmitted by the externalapparatus in accordance with a notification of the identificationinformation; recording second sets in the table other than the firstsets, the second sets each comprising the notified identificationinformation and the acquired AR data in association with each other; andsuperimposing and displaying the superimposing information correspondingto the reference image on the captured image based on a result frommatching the subject in the captured image with the reference imageincluded in the AR data which is recorded in the table, wherein eachtime the identification information is acquired, it is determinedwhether the AR data corresponding to the identification information isalready recorded in the table, and (i) if the AR data is determined toalready be recorded, the identification information is not notified tothe external apparatus and the AR data which is already recorded is notreacquired, and (ii) if the AR data is determined to not be recorded,the identification information is notified to the external apparatus,the AR data which is transmitted by the external apparatus in responseto the notification of the identification information is acquired, andthe newly acquired AR data is recorded in association with thecorresponding identification information as a new second set in thetable.